perf: lex block tokens only and reuse parsed blocks while streaming - #608

Description

parseMarkdownIntoBlocks runs on every streamed token. Two things made it slower than it needs to be:

1. It called Lexer.lex, which also runs marked's inline tokenizer. Only the raw block text is used, so it now calls blockTokens directly.
2. It lexed the whole document every time. While a response streams, text is only appended, so blocks before the last safe boundary are reused and only the rest is lexed again.

The boundary needs care. A block that ends with a blank line can still absorb the next block when that block's first line changes (2 is a paragraph, 2. is a list item), and a lone # ends the paragraph above it while #x continues it. So blocks are reused only up to the last one that ends with a blank line and is followed by at least two more blocks.

Type of Change

• Performance improvement

Changes Made

• lib/parse-blocks.tsx: call blockTokens instead of Lexer.lex; keep the last parse in one cache entry and re-lex only the tail when the input extends the previous input. Anything else falls back to a full parse.
• __tests__/parse-blocks-incremental.test.tsx: streams 40 tricky documents plus 400 seeded random ones, one, five, and varying characters at a time. Every prefix must equal a fresh full parse.
• __benchmarks__/parse-blocks.bench.ts: adds a case that streams text after a long document.
• Changeset (patch).

Testing

• All existing tests pass
• Added new tests for the changes
• Manually tested the changes

pnpm vitest bench --run __benchmarks__/parse-blocks.bench.ts, main vs this branch, Node 24 on Apple Silicon:

Code-only and HTML-only inputs have no inline content, so they do not change. The React re-render bench moves from 135 to 147 ops/s because it is dominated by React work.

Checklist

• My code follows the project's code style
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings or errors
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have created a changeset (pnpm changeset)

Additional Notes

The cache is a single module-level entry. Two documents streaming at once fall back to full parses, which is the current behavior.

One known difference: marked drops a duplicate link definition (a second [x]: url for the same label) from a full parse, so the streamed tail keeps text the full parse loses. Neither renders anything for it.